Superresolution metrology methods based on singular distributions and deep learning

ABSTRACT

Methods for determining a value of an intrinsic geometrical parameter of a geometrical feature characterizing a physical object, and for classifying a scene into at least one geometrical shape, each geometrical shape modeling a luminous object. A singular light distribution characterized by a first wavelength and a position of singularity is projected onto the physical object. Light excited by the singular light distribution that has interacted with the geometrical feature and that impinges upon a detector is detected and a return energy distribution is identified and quantified at one or more positions. A deep learning or neural network layer may be employed, using the detected light as direct input of the neural network layer, adapted to classify the scene, as a plurality of shapes, static or dynamic, the shapes being part of a set of shapes predetermined or acquired by learning.

The present application claims the priority of two U.S. ProvisionalApplication Ser. Nos. 62/551,906 and 62/551,913, both filed Aug. 30,2017, and both incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to methods and apparatus for opticalmeasurement of geometrical features, using machine, representation anddeep methods of learning, and, more particularly, for opticalmeasurement using a projected singular light distribution.

BACKGROUND ART

Metrology is the technical field of measurement. Measurements based onfluorescence, multiphoton imaging, Raman scattering, opticaltransmission, reflection or optical scattering, are practically limitedin resolution to the limit stated by Ernst Abbe in 1873. The Abbe limitarises due to diffraction by a defining aperture of the optical systemused to illuminate or collect light from a sample, as discussed below.

Some biological and life science objects can be modeled mathematicallyas elementary geometrical shapes. Precise quantification of theparameters of such a model is one of the challenges of modern biologyand life science; it will reveal hidden information, directly orstatistically, about the biological objects and their functionality.Sirat, in U.S. Pat. No. 9,250,185 (hereinafter, the “Sirat '185patent”), which is incorporated herein by reference, defines a “luminousbiological object”, as a “map” of the physical object, “in the sense of. . . general semantics.” (Sirat '185 patent, col. 18, lines 15-22.)

Measuring shape, or other geometrical attributes of an object, consistsof applying the parameterized shape assumption—referred in thisinvention as the mathematical prior—on the elemental and/or thereconstructed data from a “luminous biological object,” and retrievingthe parameters of the shape from the data. Furthermore, anothermathematical constraint exits on the data, the positivity, which is dueto the intrinsic positivity of light intensity.

In prior art, both the mathematical prior and the positivity constraintrely on a traditional imaging paradigm. The mathematical prior andpositivity constraint have to be applied on the resulting images, as apost-processing step, after image acquisition and reconstruction. Thisimaging paradigm can be traced back to the earliest days of microscopy,in which shape measurement was conceived as a concomitant of imaging andnot as a discipline per se.

The shape assumption is a strong mathematical prior, and wouldtremendously reduce the number of degrees of freedom of the solution,and create for the solution an embedded parametric space, once thepresent invention is described below. As used herein, the term“mathematical prior” shall refer to any mathematical assumption withrespect to the result of a measurement that reduces the number ofdegrees of freedom of the problem at hand. Applying the mathematicalprior after the acquisition and reconstruction processes, as has beendone in the past, with respect to the data acquired using a generalimaging paradigm, reduces the precision of the overall metrologyprocess.

Historical Perspective

After some training on newer hardware and software, Ernst Abbe, wholived from 1840 to-1905, would have been comfortable using the latestgeneration of biological microscopes for imaging and measurement. Thepersistence of biological imaging and metrology methods is due to, aboveall, the quality of the concepts developed by our predecessors, but itis also due to the difficulty to modify rooted working procedures.

However, several new trends are shaking entrenched microscopicobservation archetypes, even in biology. A major evolution results froma desire to quantify precisely elementary parameters of models ofbiological objects in a comprehensive statistical way and not to basethe biological observation only on images or assumptions.

J. Shamir, in “Singular beams in metrology and nanotechnology,” Opt.Eng., vol. 51, 073605 (2012), and in U.S. Pat. No. 7,746,469, both ofwhich are incorporated by reference, has suggested the use of singularlight distributions to quantify distributions of particle size.

To the Inventor's knowledge, the creation of null images that use theabsence of photons to achieve resolution unattainable where light itselfis imaged has never been suggested. Shamir's teachings, in particular,use elastic interactions, such as reflection, transmission and (elastic)scattering. It would be advantageous to obtain information based on nullimages, and the invention, described in detail below, illustrates howsuch imaging may be performed.

Background: Measurement of Parameters of Geometrical Shapes

While all the measuring systems and methods used in biologicalmicroscopy need not be reviewed, theoretically, the limit of measurementof any parameter, including the parameters describing a shape, isultimately dictated by signal to noise and can be quantified using theCramer Rao lower bound, CRLB, defined below. However, for shape-relatedparameters, the calculation is badly conditioned and relies onmeasurements that are not precise due to practical and experimentalconditions.

As an example, Gustafsson, “Nonlinear structured-illuminationmicroscopy: wide field fluorescence imaging with theoretically unlimitedresolution,” Proc. Nat. Acad. Sci., vol. 102, pp. 13081-86 (2005),incorporated herein by reference, presented nonlinearstructured-illumination microscopy, a super-resolution technique,coupling nonlinear saturation effects and structured illumination.Gustafsson's technique, considered exemplary by persons skilled in themicroscopic arts, demonstrated, in the best case, a resolution of 50 nm.This measurement was performed on calibrated, isolated, planar,identical manufactured beads, able to sustain high energy levels, notprone to photobleaching or phototoxicity, which were chosen byGustafsson to be of precise diameter of 51 nm, a bead of size almostidentical to the resolution limit.

Based on the foregoing data, the influence of the bead size, thesimplest possible measurement of a shape parameter, calculated by anexact procedure, (“known bead shape was removed from the reconstructionby linear deconvolution”), Gustafsson 2004 was less than 10% of the“System Ruler”, defined below, of his experiment. If the same experimentwere to have been performed as a procedure to quantify the value of anunknown diameter of the bead, the relative precision would have been 5%,a relatively poor performance in this ideal case. This indicates thedesirability of an improved tool for measurement of shape parameters,such as described below in accordance with the present invention.

Deep Learning Background

An abundant literature surrounds deep learning. The reader is referredto Wikipedia, en.wikipedia.org/wiki/Deep_learning, and to referencescited therein. A synthesis of deep learning concepts may be found inLeCun et al., “Deep learning,” Nature, vol. 521, pp. 436-45 (2015),hereinafter “LeCun 2015,” incorporated herein by reference, definingseveral basic concepts, used in the following. According to LeCun 2015:

-   -   Representation learning is a set of methods that allows a        machine to be fed with raw data and to automatically discover        the representations needed for detection or classification.        Deep-learning methods are representation-learning methods with        multiple levels of representation, obtained by composing simple        but non-linear modules that each transform the representation at        one level (starting with the raw input) into a representation at        a higher, slightly more abstract level. With the composition of        enough such transformations, very complex functions can be        learned. For classification tasks, higher layers of        representation amplify aspects of the input that are important        for discrimination and suppress irrelevant variations. An image,        for example, comes in the form of an array of pixel values, and        the learned features in the first layer of representation        typically represent the presence or absence of edges at        particular orientations and locations in the image. The second        layer typically detects motifs by spotting particular        arrangements of edges, regardless of small variations in the        edge positions. The third layer may assemble motifs into larger        combinations that correspond to parts of familiar objects, and        subsequent layers would detect objects as combinations of these        parts. The key aspect of deep learning is that these layers of        features are not designed by human engineers: they are learned        from data using a general-purpose learning procedure.        (LeCun 2015, p. 436, emphasis added)

The archetype of a deep learning network as currently practiced includesprocessing an acquired image. FIG. 6 exemplifies a deep learning networkaccording to previous art, following LeCun 2015 teaching; an image 10,illustrated by a cat picture, is the input of the network. A first layer11 typically learns the presence or absence of edges at particularorientations and locations in the image. Additional layers, representedschematically by numerals 12 and 13, learned from data using ageneral-purpose learning procedure, are able to identify the cat speciesand that it is not a dog.

“CODIM,” as used herein, refers to COnical DIffraction Microscopyhardware, as described in Caron, et al. (2014). “Conical diffractionillumination opens the way for low phototoxicity super-resolutionimaging.” Cell adhesion & migration, vol. 8, pp. 430-439 (2014),hereinafter “Caron 2014,” incorporated herein by reference. Use ofhardware as described in Caron 2014 is referred to as “the imagingcase.” The hardware set-up and use described below as “the metrologycase,” is not part of the prior art. In the imaging case, singular lightdistributions are projected on a regular grid, the grid being Cartesian,or on a grid optimized as a function of the retrieved data.

The canonical way to apply CODIM in a deep learning or machine learningnetwork is to reconstruct the image and to apply a known algorithm tothe reconstructed image. The limitation of such methods is the timeburden and information corruption entailed in image reconstruction.

The Sirat '185 patent states:

-   -   The measurement system will calculate an evaluation of the        descriptors of the fluorophores, the measured map. This measured        map differs from the original map, due to noise, measurement        conditions, the system limits or measurement uncertainty. This        information map can be developed later into different levels of        abstraction.        (Sirat '185 patent, col. 18, lines 37-42, emphasis added)

The Sirat '185 patent further teaches that:

-   -   The primordial Information, the map in the terminology of        general semantics, is the set of descriptor fluorophores and        their evolution over time. Biological and geometric information        will only be extrapolations of this primordial information. The        measurement system will calculate an evaluation of the        descriptors of the fluorophores, the measured map.        (Sirat '185 patent, col. 18, lines 32-37)

Sirat '185 assumed that “[t]he measurement system will calculate anevaluation of . . . the measured map” (Ibid., emphasis added), seeingthe measurement map as a prerequisite to any information gathering. Inrequiring a measurement map as a prerequisite to information gathering,Sirat '185 was following the prevalent consensus that the “measured map”is a prerequisite to further processing, as geometrical and biologicallevels of abstractions, described by Sirat '185, in the same paragraph.

SUMMARY OF EMBODIMENTS OF THE INVENTION

In accordance with embodiments of the present invention, methods areprovided for determining a value of an intrinsic geometrical parameterof a geometrical feature of a specified dimensionality characterizing aphysical object. The method has steps of:

-   -   a. projecting a singular light distribution characterized by a        first wavelength and a position of singularity onto the physical        object;    -   b. detecting light excited by the singular light distribution        excited that has interacted with the geometrical feature and        that impinges upon a detector, the light constituting detected        light;    -   c. identifying and quantifying a return energy distribution at        one or more positions of the singular light distribution as a        quantification of the intrinsic geometrical parameter; and    -   d. determining the value of the intrinsic geometrical parameter        based on the parameters retrieved from energy distribution.

In accordance with other embodiments of the present invention,successive applications of the aforesaid method are performed, in such amanner as to retrieve a single or a set of intrinsic geometricalparameters. The intrinsic geometrical parameter may be one of the sizeand ellipticity of a point-object. A measure of displacement may bedetermined that is an offset transverse to a line.

In accordance with further embodiments of the invention, the intrinsicgeometrical parameter may be a width of a line. The intrinsicgeometrical parameter may be based upon a model shape of the geometricalfeature. Detecting light may include employing a pixelated detector.

In accordance with another aspect of the present invention, a method isprovided for representation learning for classifying a scene into atleast one geometrical shape, static or dynamic, quantified by anadequate set of parameters, each geometrical shape modeling a luminousobject. The method has steps of:

-   -   a. projecting a singular distribution of light onto a scene;    -   b. detecting a light distribution, reemitted by the scene upon        illumination by the singular light distribution that has        interacted with each luminous object and that impinges upon a        detector, the light detected constituting detected light;    -   c. measuring at least one projection of a singular distribution        at a given position to obtain a set of measurements with respect        to a scene; and    -   d. employing a deep learning or neural network layer, using the        detected light as direct input of the neural network layer,        adapted to classify the scene, as a plurality of objects,        physical or biological, or as geometrical shapes, static or        dynamic, the objects or shapes being part of a set of objects or        shapes predetermined or acquired by learning.

In accordance with other embodiments of the present invention, the atleast one geometrical shape may be either dynamic or static. The methodmay also include acquiring data that is imaging data acquired by CODIMhardware, or that is metrological data acquired by Metrology Applicationhardware.

In accordance with further embodiments of the present invention, themethod may have a further step of acquiring data that is imaging dataacquired by CODIM hardware, wherein resulting classification informationis used to drive a metrological hardware to implement the metrologicalmethod. A further step may include providing control hardware and acontrol procedure to drive a metrological hardware to implement themetrological method. A further step yet may include feeding data to theneural network that has been acquired at different times and including atime dependence of luminous objects in a neural network recognitionprocess.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of the invention will be more readily understoodby reference to the following detailed description, taken with referenceto the accompanying drawings, in which:

FIG. 1 shows a prior art modulation transfer function (MTF) of a CODIMlight distribution compared to an Airy pattern.

FIGS. 2A-2D represent an Influenza Type A virus (IAV) used in thepresent Description as an illustrative example, with the virusrepresented schematically in FIG. 2A, and with its three-dimensionalmodel represented in FIG. 2B and its two-dimensional model beingrepresented in FIGS. 2C and 2D, respectively, for an isotropic andnon-isotropic model.

FIG. 3 is reproduced from U.S. Pat. No. 9,250,185 (hereinafter “Sirat'185”) and presents some of the different singular distributionsavailable using conical diffraction; and

FIG. 4 represents a schematic of the locus of a crescent moon, known tobe a spiral and plane A and B crossing the spiral at two points withdifferent lateral positions, in accordance with an embodiment of thepresent invention.

FIG. 5 is a flowchart showing steps of a method of representationlearning in accordance with an embodiment of the present invention;

FIG. 6 schematically depicts the implementation of a deep learningsystem according to prior art.

FIG. 7 schematically depicts the implementation of a deep learningsystem on a CODIM imaging system.

FIG. 8 schematically depicts the implementation of a direct deeplearning network on a CODIM imaging system according to the presentinvention.

FIG. 9 schematically depicts the implementation of a direct deeplearning network on a compound imaging system according to the presentinvention.

FIG. 10 schematically depicts the implementation of a direct deeplearning network on a general object, represented for a heuristic reasonas a cat. The fact that the intermediate images are represented assub-images of a part of the cat picture is only a graphical artifice.The information will be unstructured information and not partial view ofthe image.

FIG. 11 depicts the implementation of a controlled direct deep learningnetwork on a CODIM imaging system according to the present invention.

FIG. 12 depicts the implementation of a controlled direct deep learningnetwork on a general object, represented for a heuristic reason as acat. The fact that the intermediate images are represented as sub-imagesof a part of the cat picture is only a graphical artifice. Theinformation will be unstructured information and not a partial view ofthe image.

DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

New methods are presented herein, in which the measurement procedure isa separate, dedicated procedure and architecture, initialized in somecases by an imaging procedure, but fully embedded in a separate tool,the metrology tool.

The invention described below is directed towards accurately measuringgeometrical features that are smaller than the diffraction limit oflight employed in their measurement, with minimal flux.

This invention is especially adapted to measuring geometrical featuresusing inelastic light interaction, as, but not limited to, fluorescence,multi-photon imaging, or Raman scattering, in which the emerging lightcan be separated by simple means from the incoming light.

Some embodiments of the present invention relate to methods, and tohardware systems implementing them, for machine learning of geometricaland functional features and objects. These methods entail classificationand recognition of features and objects; these methods differ from theactual solutions based on an imaging paradigm and on further processingsubsequent to imaging.

While some embodiments of the present invention is primarily describedwith reference to systems and methods implemented in fluorescentmicroscopy, it is to be understood that extension to reflected orscattered light is straightforward and within the ken of persons ofordinary skill in the art who have understood the present description.Moreover, while this invention is described herein in terms of systemsand methods for metrological tools of fluorescent objects encountered inbiology, they may be readily applied to other imaging modalities and toother disciplines and domains, such as, but not limited tosemiconductors and machine vision.

It would be useful to apply the mathematical prior before thereconstruction process and embed it in the acquisition process, howeverthere has never been any teaching how that might be done, and that isnow described here in detail. The application of a mathematical priorbefore image reconstruction, in accordance with embodiments of thepresent invention, may advantageously increase the accuracy of themeasurement of the parameters of the shape, reducing the number ofobservations required for measurement and increasing the stability andreproducibility of the overall process.

Methods described in accordance with embodiment of the present inventionrely on the specificity of Poisson's law, explicitly the absence ofphoton noise—up to a negligible number due to quantum physics theory—inthe absence of photons, any photons, both incident and emitted on thedetector, and not a posteriori correction. Such methods require amechanism to physically remove the incident light, through a filteringoperation. It cannot be a mathematical post-processing operation becauseif the noise, have been already introduced in the system, it cannot befiltered out by simple means. This is the advantage and specificity ofinelastic light interactions; inelastic light interaction, will refer inthis invention to any light interaction able to create (or emit) newphotons with a physical characteristic differentiating them from theincident photons in a way that the emitted photons can be physicallyseparated from the incident photons. It is assumed that a “filter”allows separating the emitted photons from the incident photons. Theobvious example of an inelastic light interaction is an interactioncreating photons at a different wavelength; in this case the filter willbe a wavelength sensitive filter. Examples of inelastic lightinteraction are fluorescence, multi-photon imaging or Raman scattering.Elastic light interactions, such as reflection, transmission or elasticscattering, do not allow for such discrimination in a simple way.

As explained hereinafter, embodiments of the present invention rely oncreating null images, or images close to the null image. Such imagesallow additional resolution, theoretically unlimited, as will bedescribed below, by combining a loophole of Abbe's law with the absenceof noise due to Poisson's law. A prerequisite is the absence of anyspurious photon, required to fulfill the theoretical conditions. Suchconditions may be met almost only using inelastic light interactions, asfluorescence, multiphoton interactions and Raman scattering, in whichthe incoming beam can be totally filtered by spectral means.

For heuristic convenience, the invention may be illustrated herein by ahypothetical example; this example is used just as an illustration andis not intended to represent a tested experimental case. A particularbiological object is assumed to constitute a luminous object, and thisbiological object, for illustration, is assumed to be an Influenza TypeA virus, (IAV), represented schematically in FIG. 1A. The IAV virus isknown to have a typical size between 80 to 120 nm, and, as discussed byBadham et al., “The biological significance of IAV morphology in humanclinical infections is a subject of great interest,” in FilamentousInfluenza Viruses, Current clinical microbiology reports, vol. 3, pp.155-161, (2016), incorporated herein by reference.

The IAV virus is assumed to be marked uniformly in all its volume withappropriate fluorescent proteins. Three different models can be used todescribe the virus:

-   -   a. In the imaging paradigm used in prior art, due to the small        size of the virus, under the diffraction limit, the best model        will be to represent the virus as a luminous point, emitting        fluorescent light; it will indeed allow accurate monitoring of        the virus position.    -   b. Alternatively, using for example an electron microscope with        nm or sub-nm resolution, various details of the virus may be        characterized, including the hemagglutinin 201 and        neuraminidases 203.    -   c. In accordance with teachings of the present invention, an        intermediate way is employed in which the virus is described as        a point-object, defined below, modeled either in three        dimensions as a sphere, (FIG. 2B), or in two dimensions as a        uniform circle, (FIG. 2C) with given radius, or as an ellipse        (FIG. 2D), with parameters being the half-minor axis, a, the        half-major axis, b, and the angle of the major axis, θ, relative        to a Cartesian reference frame.

In accordance with further embodiments of the present invention, deeplearning may be employed, as described in detail below.

Methods in accordance with embodiments of the present invention mayadvantageously provide for measurement entailing resolutions greaterthan the diffraction limit imposed by the optics and minimal photonfluxes.

In accordance with embodiments of the present invention, one or moremathematical priors resulting from the shape assumption are embedded ina specific set of singular light distributions, allowing full advantageto be taken of the properties of singular light distributions. Thiscoupling is based on dedicated metrology tools, relating themathematical prior and the set of distributions; in other words, themeasurement of a specific shape, defines mathematically a prior and asolution domain, which itself determines the characteristics of theacquisition tool and the type of distributions required.

Definitions: As used herein and in any appended claims, the followingterms will have the following specified meanings, unless the contextrequires otherwise:

The term “value” as used herein and in any appended claims shall referto a real number characterizing a quantity associated with a parameter.It is to be understood that, in a practical context, the quantityassociated with a parameter may be characterized within some range,constituting the accuracy of a measurement. In that case, the term“value” may be used as shorthand for a distribution of values.

the “System Ruler”, is a value, which is considered by a person ofordinary skill in the imaging arts to characterize the capacity of thesystem to discerned details.

In imaging systems, the “Rayleigh criterion”—the capacity to separatetwo adjacent points—is the generally accepted criterion for the minimumresolvable detail, even if the observed FWHM of a point or a line is, inmany cases, used as a practical evaluation of the “diffraction limit”, aqualitative term used commonly to quantify the minimum resolvabledetail. In this invention, we use the FWHM of an infinitely thin line asthe system ruler. A value will be negligible if it is “much smaller”than the System Ruler, where “much smaller” is defined as smaller by afactor of 3 or more.

The term “Abbe's resolution limit” as used herein is as found inSchermelleh et al., “A guide to super-resolution fluorescencemicroscopy,” J. Cell Biology, vol. 190, pp. 165-75 (2010), hereinafter“Schermelleh 2010”, incorporated herein by reference:

-   -   Abbe's famous resolution limit is so attractive because it        simply depends on the maximal relative angle between different        waves leaving the object and being captured by the objective        lens to be sent to the image. It describes the smallest level of        detail that can possibly be imaged with this PSF “brush”. No        periodic object detail smaller than this shortest wavelength can        possibly be transferred to the image.

The expression “above the Abbe's limit” is defined to refer to an objectcontaining periodic structures containing details smaller than anydetails of the System Ruler, thus limited by the Abbe's limit. Therationale of this definition is that such an object contains spatialfrequencies above the Abbe's circle of frequencies in the apertureplane.

In estimation theory and statistics, the Cramér-Rao bound (CRB) or,equivalently, the “Cramér-Rao lower bound (CRLB)”, expresses a lowerbound on the variance of estimators of a deterministic (fixed, thoughunknown) parameter. The precise definition employed herein is asprovided in en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound, whichis incorporated herein by reference.

A “localized” light distribution, as the term is used herein, shallrefer to a light distribution with energies concentrated on a smalldomain. A light distribution will be localized if the energies, outsidea radius of 3.5*the half Rayleigh criteria are substantially zero.

This invention description assumes that the optical system described isclose to being “photon noise limited,” as described inen.wikipedia.org/wiki/Shot_noise, or is close to being photon noiselimited, i.e. the Gaussian noise component is smaller than half thephoton (or shot) noise. The optimal case is indeed a “photon noiselimited” optical system as described and a “Gaussian noise limited”system will collect only part of the advantages of this invention, butis still in the scope of this invention.

“Full width at half maximum” (FWHM) is an expression of the extent of afunction given by the difference between the two extreme values of theindependent variable at which the dependent variable is equal to half ofits maximum value, from Wikipedia,en.wikipedia.org/wiki/Full_width_at_half_maximum.

The “dimensionality” is defined as any one of the three physical orspatial properties of length, area and volume. In geometry, a point issaid to have zero dimension; a figure having only length, such as aline, has one dimension; a plane or surface, two dimensions; and afigure having volume, three dimensions.

The dimensionality of a geometrical feature shall refer to thedimensionality, of a corresponding idealized feature in the limit inwhich the extent of the geometrical feature (such as the ‘diameter’ of apoint object, or the ‘width’ of a line or the ‘thickness’ of a coating)is much smaller than the extent in any other dimension which tends to bezero.

A “locus”, as the term is used herein, shall denote a set of points(commonly, a point, a line, a line segment, a curve or a surface), whoselocation satisfies, or is determined by, one or more specifiedconditions.

A “geometric parameter” shall refer to any measurable quantitycharacterizing an extent within a physical space. Thus, for example, adistance (defined in three-dimensional Euclidean space or along asurface) is an example of a geometrical parameter. An area defined on asurface is another example of a geometrical parameter.

A “geometrical feature” is a physical object characterized or modeledgeometrically as a locus, quantified by one or more geometricalparameters; for example, the thickness of a line, describes a locus, theline, which is a locus, or the position and thickness of a line segment,which is also a locus. The “embedding space” represents the parametricspace spanned by the geometrical parameters.

A “point”, is a geometrical feature in two or three dimensions with zerodimensionality and zero size. It is an overstated simplification,erasing much information on real objects, but simplifying tremendouslythe assumptions. We refer to an object with small but not negligiblesizes, compared to the System ruler, in all the three dimensions as“point-object”. The terms small or negligible has to be appreciatedcompared with the system ruler, which, for optical systems is the limitof diffraction. A point object is determined by its position and itssize, which can be isotropic, or not, in two- or three-dimensions. Mostbiological objects are, in system rulers of diffraction limited orsuper-resolved optical systems point-objects, and the a priori dismissalof the information carried by the point-object is a tremendous loss. Thedifferentiation between points and point-objects is of major importancein this invention.

A “line” (and similarly other terms that refer to shapes that areone-dimensional in theory . . . ) shall refer to a geometrical feature(i.e., to a physical object, having a length, width and thickness),where the length is at least 10 times either the width or the thickness.A “line object” is defined following the same concepts as of apoint-object.

An “intrinsic geometric parameter,” as the term is used herein and inany appended claims, shall refer to a parameter characterizing aproperty of a surface (in n-dimensional space) that is independent ofthe isometric embedding of the surface in Euclidean space. Thus, forexample, the width of a “line” though a given point is an intrinsicparameter of the line, independently of any direction of the line inspace

Referring to a geometrical element in an abstract space, a set ofparameters shall be referred to as “adequate” if and only if therepresentation of the geometrical element described by the set ofparameters faithfully represent the geometry class and parameters of theobject in the limit of the measurement uncertainty.

“Singular Optics”, which includes “optical vortices” as its simplestexample is today an emerging domain of optics, with theoretical as wellas practical applications. Detailed description may be found in Nye, etal., Dislocations in wave trains. Proceedings of the Royal Society ofLondon A, vol. 336, pp. 165-90 (1974), and in Soskin, et al., “Singularoptics,” Progr. in Optics, vol. 42, pp. 219-76, both of which referencesare incorporated herein by reference.

“Inelastic optical interaction” refers to interactions between light andmatter creating photons which differ in wavelength from the incomingbeam. Inelastic optical interaction includes, but are not limited tofluorescence, multiphoton interactions and Raman scattering.

The “locus of a singular distribution” is the ensemble of Cartesianpositions on which the intensity of the singular distribution is zero.The locus of a singular distribution defines a family of elementaryshapes, which, with adequate parameters, “the nominal parameters” andpositioned at the right position; “the nominal position” will not emit(or reflect or scatter) light. In this case we will coin the new conceptand express that the “singular light distribution embeds the shape”. Forexample, for the light distribution depicted as case 10 (or 01) in FIG.2 , the vortex, the locus is an infinitesimally small point and the onlyfamilies of shapes which can be studied using a vortex are the pointshape and the point-objects shape, which will also create no energy witha radius of value zero. For the distribution described as case 42 (or24) in FIG. 2 , named “vertical half-moons”, the relevant families ofshapes are more diverse, including lines and line-objects but alsopluralities of the point shapes and of the point-objects. Additionalcases will be developed in the following.

Conical refraction is an optical phenomenon predicted by Hamilton,“Third supplement to an essay on the theory of systems of rays,” Trans.Royal Irish Acad., vol. 17, pp. 1-144 (1837), and experimentallyconfirmed two months later by Lloyd, Lloyd, H. (1831). “On the PhenomenaPresented by Light in Its Passage along the Axes of Biaxial Crystals,”Philos. Mag., vol. 1, pp. 112-20 (1831). Both of the foregoingreferences are incorporated herein by reference. Conical refractiondescribes the propagation of a light beam in the direction of theoptical axis of a biaxial crystal. Hamilton predicted that the lightemerges in the form of a hollow cone of rays.

A description, provided in Berry, “Conical diffraction asymptotics: finestructure of Poggendorff rings and axial spike,” J. Opt. A: Pure andApplied Optics, vol. 6, pp. 289-300 (2004), which is incorporated hereinby reference. A prior art system based on conical diffraction for superresolution microscopy is described in Sirat 2016.

U.S. Pat. No. 8,514,685 (Sirat '685) described specific properties ofconical diffraction in thin crystals and demonstrated the use of conicaldiffraction to shape optical beams. A thin biaxial crystal transformsthe Point Spread Function (PSF) of a regular incident beam into anextensive family of light distributions, the choice of the distributionbeing controlled by the input and output polarizations. Practically,beam shaping is achieved by enclosing the biaxial crystal betweencontrollable polarizers; this simple optical set-up, similar to apolarimeter, has the ability to switch from one pattern to anotherpattern with a different topology in microseconds—or even faster. Inaddition, these patterns are perfectly co-localized, as they areproduced by the same primary optical beam.

The CODIM beam shaper may be used as an add-on to a scanning module andthe distributions are scanned on the sample, yielding severalmicro-images for each scan point. A CODIM system is available fromBioAxial SAS of Paris, France.

The CODIM beam shaper generates compact, localized light distributionsusing the conical diffraction principle. Each microimage contains alarge amount of high frequencies, close to Abbe's limit (up to a factorclose to 3 compared to an Airy pattern 101, as depicted in FIG. 1 .These light distributions, projected onto a region of the samplereferred to as a “scene,” are analyzed using algorithms well known topersons of ordinary skill in the art. For exemplary algorithms suitedfor deriving super-resolved imagery from microimages, the reader isreferred to Published Application WO 2016/092161 (published in Englishas US 2017/0336326, hereinafter “the Sirat '161 application), which isincorporated herein by reference. This allows reconstruction of asuper-resolved image, for general objects, with an improvement ofresolution up to a factor of 2. Additionally, these algorithms,leveraging positivity constraint and sparsity, allow for the resolutionto be improved even further, for adequate samples.

Ultimately, the conjunction of much lower distribution peak power, useof a high quantum yield camera and a longer camera exposure timedrastically reduce the peak power—and the energy—of light sent to thesample. Such is the reason for this method's very low photobleaching andphototoxicity. This also avoids fluorophore saturation issues, makingthe method linear and quantitative.

In the present Description, the term “energy law” is defined as follows:Assuming that an object has been modeled as a mathematical abstraction,the geometrical shape, the “energy law” is the parametric relationbetween the energy, as a function of the shape parameters and theposition. It creates a relationship quantifying the energy dependence ofthe parametric space. The energy law may include the energydistribution, emitted by a luminous object with a shape identical to thegeometric shape.

In the present Description, it is assumed that the optical singulardistributions can be controlled, in such a way to switch from one typeof distribution to another, from a predetermined family ofdistributions, and to modify the parameters of the optical singulardistributions using external means, as described in Sirat '185. Othersolutions exist, not requiring the optical singular distributions to becontrolled, and are part of this invention, but they will be much morecumbersome.

Imaging performed using methods in accordance with the present inventionmay be referred to as “compound imaging,” and may also be referred to ascoded imaging, as defined in Wikipedia, en.wikipedia.org/wiki/Codedaperture, or as “indirect imaging,” in reference to imaging processes inwhich the elemental data from the scene is not an image, or a map, inthe sense of Sirat '185 patent, describing the object. In a compoundimaging process, the data measured by the physical instrument, referredto as the “intermediate result,” (the term used in the Sirat '161application), or elemental (or underlying) data, as used synonymously inthe present Description, contains information from which a reconstructedimage, can be created through a reconstruction process. We will identifyin this invention a compound imaging process as an imaging process inwhich it is not possible to assume an intensity, or a number of photons,measured at a specific time and position to a specific pixel of an imageor to a specific attribute of an element in a map. In simplified words,a compound imaging system the intensity or number of photons does nothave a direct translation to pixel data but contains informationrelevant to several of them.

In the domain of superresolution microscopy, CODIM imaging is an exampleof a compound imaging process as well as Structured IlluminationMicroscopy (SIM), or localization microscopy techniques such asPALM/STORM, referenced in Sirat '185 patent.

The “measurement procedure”, will consist of a set of “elementarysingular measurements”, each elementary singular measurement consistingof the projection of a predetermined singular light distribution, withgiven parameters, positioned such that the locus of the singulardistribution is embedding the geometrical shape at the nominalparameters, and of a “light measurement” quantifying the light incomingfrom the object, its intensity and optionally its spatial distribution,as recorded on a light detector, either pixelated or not. In some cases,“elementary regular measurements” consisting of the projection of apredetermined regular light distribution, may also be used, to completethe information gathering.

The “inputs” of a measurement refer to the position of the opticalsingular distribution, its type and parameters. The “outputs” are thelight intensity, and/or the light distribution, recorded in themeasurement. We assume that the reader is a man skilled in the art andis familiar with “Control theory”. Control theory is aninterdisciplinary branch of engineering and computational mathematicsthat deals with the behavior of dynamical systems with inputs, and howtheir behavior is modified by feedback (from Wikipedia).

The “control means” will refer to a set of control hardware, able tomodify the inputs and a “control algorithm”, able to foresee next stepsof input values required to quantify directly or by successiveapproximations the “energy law” in a way adequate for retrieving theparameters with precision. The “inverse energy law” is a recipe, optimalor not, able to retrieve the parameters of the shape, from a set ofmeasurements of a single singular distribution or of a set of singulardistributions. It is embedded in the control algorithm. It will bechosen to optimize the functional parameters of the system, either thenumber of steps required, the overall energy—or power—impinging on thebiological object, the speed of measurement, any combination of theabove or any other functional parameters of the system.

In the context of singular distributions, a “value close to zero” shallrefer to energy used to qualitatively describe intensity projected orenergy emitted which are reasonably smaller than the maximum intensityavailable on the projected light or of the energy emitted if the maximumof the projected light is impinging on this point. A quantitative valuefor a close to zero intensity or energy is a factor of six between theintensity projected and the maximum intensity of the distribution orbetween the energy emitted and the energy emitted if illuminated atmaximum intensity. It is worth mentioning, that assuming Poisson noise,energy close to zero will have a noise value markedly smaller, close totwo and half times less, then at maximum energy. Similarly, a parametervalue of geometrical shape “close to zero” will have a value smallerthan the full range of the parameter divided by 2.5.

In accordance with embodiments of the present invention, an observedobject may be modeled either as a specific shape, known or hypothesizedbeforehand, or either as a shape in a list of potential shapes. Thegeometrical shape may be described in terms of:

-   -   a. a spatial position, the “position” of the shape, referring to        the position of an origin point determined in the shape, in most        cases, the center of gravity of the light distribution.    -   b. a set of “structural parameters” describing the “shape        parameters”, for example, for an isotropic circle its radius        (FIG. 2C), for an object represented by an ellipse (FIG. 2D),        the ellipse semi-axes axes and angle.

The term “machine learning,” as used herein, refers to a field ofcomputer science that uses statistical techniques to give computersystems the ability to “learn” (e.g., progressively improve performanceon a specific task) with data, without being explicitly programmed to doso.

The term “representation learning” and “deep learning” are used inaccordance with LeCun 2015, as laid out in above. LeCun 2015 clearlydraws the borders between “representation learning” and “deep learning”on one hand and standard “machine learning”, on the other.

The terms “representation learning” and “deep learning” are usedinterchangeably herein because all embodiments of the present inventionapply to both. Parts of the invention may be applied also to machinelearning.

In the Metrology Case (as defined above), data are acquired at nominalpositions of the objects identified. A “nominal position,” as it refersto an object, shall mean a position assumed from previous knowledge orfrom a hypothesis to be the position of the object.

It is assumed, for purposes of the present description, that a separatemechanism had been used to gather the nominal position of the object.Within the scope of the present invention, this mechanism may use anylocalization technique, as for example the measure of the centroid ofthe light distribution created by the object in another imaging modalityor directly on a camera.

Rationale of the Invention

The present invention introduces new methods for the measurement ofparameters of biological, life science or physical objects modeled asgeometrical shapes. In accordance with the methods described herein indetail, systems are created to directly access parameter values ofelementary geometrical shapes; the method and systems take advantage ofthe properties of singular light distributions, and are limited neitherby diffraction, as is imaging, or photon flux, as is parameterassessment using an imaging paradigm and image processing tools.

Novel measurement methods described herein are correlated with physicalinsight, first described here, to our knowledge, in the form of aspecific restriction of the direct application of Abbe's resolutionlimit to measurement.

Indeed, going back to Abbe's resolution law as described by Schermelleh2010 or by Horstmeyer, et al., “Standardizing the resolution claims forcoherent microscopy.” Nature Photonics 10, pp. 68-71 (2016),incorporated herein by reference, all “bandwidth extrapolation”techniques, reconstructing information above the Abbe's limit relies onprior knowledge, assumptions or even guesses, which in some case may bewrong. We describe here, the only case to our knowledge, ofmeasurement—without prior knowledge, assumption or guesses—above theAbbe's resolution law, which is the case . . . of a black (null) image.

Due to the positivity constraint, a null image contains no signal and sono spatial frequencies below and above the Abbe's limit and all thefrequency contents are fully known . . . to be zero; a null image allowsquantifying high frequency contents, to zero, above the Abbe's limit,from a diffraction limited optical measurement.

This is antagonistic to the naïve view of Abbe's law, which isunderstood as if that no information can be retrieved by an opticalprocess above the Abbe's limit. The case described in this invention,even if it is very peculiar, allows gathering information above thelimit, theoretically up to infinity, and so, is diffraction unlimited.

This condition is specific to optics, and similar unipolar signals, andis due to the positivity constraint, which is an additional physicalconstraint on the solution, independent of the diffraction limit. Theseconcepts will not apply swiftly to electrical signals, or similarbipolar signals. The positivity constraint is known, somehow, to allowadditional information, but its contribution to the extension of theinformation is, in most practical cases, marginal and not wellcharacterized or quantified. In this invention, the influence ofpositivity constraint is clear and determinant, allowing accuratequantification of the spatial frequencies below and above the Abbe'slimit.

Several formulations of the Abbe's resolution law coexist. A preferredphrasing of Abbe's resolution law is that of Schermelleh 2010: “Noperiodic object detail smaller than this shortest wavelength canpossibly be transferred to the image”. (Schermelleh 2010, at p. 166) Theresults of the present invention constitute a corollary of Abbe'sresolution law, and not an exception.

However, the results of the present invention are indeed an exceptionand a rebuttal of some widespread formulations of Abbe's resolution lawin the scientific literature, less carefully phrased, “no frequencycomponent can be measured above the Abbe's resolution limit”.

In most cases, the null image contains no information and this singularpoint or exception is useless and trivial and this discussion in thesecases may seem pointless and empty. However, the recorded image of anisolated point, with given size, illuminated by an optical vortexpositioned at the nominal position, will be identically zero, only ifthe size is zero. In this specific case, the full nulling of the imageand of all its frequency components is a powerful measure, from which itis possible to retrieve not only the position but also the size, as willbe explained later, of a point-like object.

This invention additionally introduces a new method, which can beapplied to any compound imaging system and is exemplifies in moredetails in the case of CODIM imaging. It consists of applying a directbridge, in most cases a specific neural network directly on theelemental data. In previous paragraph we described how to apply anatural pathway (or bridge) between elemental data in a compound imagingprocess to direct Metrology methods. In this paragraph, we describe howto apply natural pathway (or bridge) between elemental data in acompound imaging process, or to deep learning, by applying a deeplearning layer, supervised or unsupervised, directly to the elementaldata, without any image reconstruction algorithm. Additionally, thislayer will be able to collect data from measurements from several times(consecutive in most cases). The results of this layer, either themetrology or object layers, will be the direct input to furtherprocessing, bypassing the image or map steps, which may still be usedfor visualization, qualification and monitoring.

Both processes, direct metrology methods and deep learning can also becompounded to create even stronger and more powerful acquisition tools.

The methods described in accordance with the present invention differfrom all prior art methods which have ever been described or suggested,at least in that all prior art methods treat image gathering as amandatory step.

The Sirat '185 patent nowhere suggested employing a deep learning orneural network layer, using detected light as direct input of the neuralnetwork layer, able to classify the scene, as a plurality of shapes,static or dynamic, the shapes being part of a family of elementaryshapes predetermined or acquired by learning. In accordance with thepresent invention, a deep learning layer for structuring directly theraw data information as elementary objects, with metrologicalcharacteristics and attributes, these elementary objects being theadequate substrate of further generalizations. This approach, consistentwith the viewpoint of deep learning, is developed in this invention.

It is to be noted that the Sirat '185 patent relies on the physicalreality, and to define the measured map as the descriptors of thefluorophores, this new positioning is coherent with the biologicalreality, defining biological entities, as in our example, the InfluenzaA virus, as the structuring concept.

Additionally, the method can be extended to dynamic characteristics,retrieved either as time-dependent data, or preferably as higher-leveldynamic parameters. In short, getting back to our example, theintermediate layer may be able to determine the presence of apoint-object of 110 nm, moving along a path at fixed velocity, withoutmorphological changes; and we will assume, in the next layer, the objectclassification layer, that it is the Influenza A virus.

An additional merit of the proposed architecture is to avoidreconstruction artifacts by bypassing the reconstruction step. Even thebest reconstruction algorithm may be prone to artifacts and the possiblepresence of artifacts is one of the barriers to adoption of superresolution techniques by biologists. To make things even worst, manyartifacts can be mistakenly considered as small points or objects, andclassification and quantification algorithms can be spoiled by theseartifacts. Avoiding reconstruction and performing classification andquantification on the raw data totally remove this source of error.

Information above the diffraction limit exists in the metrology data, asdescribed above, and can also be present in the imaging data. Thispriceless information is typically erased by reconstruction processes,unable to process this information separately from all other informationand so forced to apply general procedures, optimized for averaged data,but not fitted to these specific cases.

Geometrical and Biological Information Through a Non-Imaging Process

The Sirat '185 patent failed to suggest how geometrical and biologicalinformation can be learned through a non-imaging process. In such anon-imaging process, the higher levels of information are gathereddirectly from the measured elemental data, using a general-purposelearning procedure, without the requirement to create (or reconstruct)an image, or in Sirat '185 patent terms, to quantify the map. Sirat '185patent fails to understand that the non-imaging direct learning might beretrieve information with much higher precision, as explained below,high computational efficiency, and without loss of data. Furthermore,such a learning procedure may have the ability to use of data present inthe absence of light, described below and referred to as the Abbe's lawloophole.

General Methods

In accordance with typical embodiments of the present invention, thefollowing features are salient:

-   -   distribution of light, at nominal position; the wavelength of        the projected light is chosen such that it is able to create an        inelastic light interaction with the light-responsive material,        the singular distribution being chosen such that it embeds the        geometrical shape, and so, with nominal parameters, the energy        measured from the light created by the inelastic interaction of        the projected light and the light-responsive material will be        zero.    -   an algorithm, “the inverse energy law,” is applied to retrieve        the parameters from the energy measurements.

Methods in accordance with the present invention may advantageouslyemploy measurement hardware and a measurement procedure controlling,dynamically or not, the sequence of measurements in order to gather theset of measurements required to retrieve the parameters of the shape.

The light-responsive material may be exogenous, based, for example bybinding fluorescent binding or endogenous, using, for example directfluorescent properties of some protein, or using Raman scattering.

For heuristic convenience, we follow each general statement of themethod, in high-level language above, by the illustration of an example;the example is the simultaneous measurement, for a point-like object, ofits position and size; to make it more illustrative, the point-object isdescribed in terms of an Influenza Type A virus, as explained in aprevious paragraph.

A general method may be illustrated using the simplest case of thevirus, modeled as a two-dimensional uniform circle shown in FIG. 2C. Thebiological object is the virus; the measurement goal is, in thisexample, to quantify the diameter of the virus between 80 to 120 nm. Thevirus is assumed to be positioned on a two-dimensional surface.

The biological object is the virus modeled as a point-object,represented as in FIG. 2C, by its position and the radius R of thevirus. The measurement consists of projecting a vortex of light, able toexcite fluorescent molecules positioned on the virus, which embeds thepoint-object geometrical shape, at several positions (x_(i), y_(j)),using a measurement hardware consisting of a galvanometrictwo-dimensional optomechanical system able to scan the vortex on thebiological sample, measuring the energy on a light detector, andconverging to the position with minimal energy, using either ananalytical or an iterative algorithm. In this case, the inverse energylaw is very simple and the position is the position of the minimum andthe energy, at minimum, is quadratically dependent on the radius(equation 3).

In general, methods of measurement in accordance with the presentinvention allow retrieving zero-one- and two-dimensional loci, using oneor more different singular distributions and retrieve simultaneously oneor more simple parameters and more complex or compound parameters.

Measurement of the Size and Position of a Point-Object

A first embodiment of the method is presented for measuring shapeparameters: measuring the position and the size of a point-object intwo-dimensions. It is one of the most common cases in the measurement ofbiological objects. Furthermore, it can be extended, mutatis mutandis tothe measurement of the position and thickness of a line, anotherwidespread task in measurements of biological objects.

Assuming a luminous object of circular shape (shown in FIG. 2C), withradius R, positioned at the origin, with a uniform density offluorophores, represented as n_(D), and assuming that N_(E)=n_(D)πR², isthe total number of emitting fluorophores, an optical system isprovided, such as that described in detail in Sirat '185 patent, that isadapted to move the position of a vortex, given as (v, 0), with highaccuracy, in the plane. In the polar coordinate notation “(m, n)”employed herein, m refers to a radius vector in a specified transverseplane with respect to the z-axis, while n refers to the angle between apredetermined axis (x in FIG. 2C) and the radius. Without loss ofgenerality, due to the symmetry of the problem, we assume that thevortex is positioned on the x-axis, i.e. the angle is zero.

The intensity of the vortex as a function of position is quadraticallydependent, through a coefficient i_(s), on the distance between thevortex null and the position where the intensity is measured. Moreparticularly, for v>R (where R is the radius of the luminous object),the energy law will be given by

$\begin{matrix}{I_{vR} = {{I_{v} + I_{R}} = {\int_{0}^{2\pi}{\int_{0}^{R}{i_{s}\mspace{14mu}{n_{D}\left\lbrack {\left( {v - {r\mspace{14mu}\cos\mspace{14mu}\vartheta}} \right)^{2} + \left( {r\mspace{14mu}\sin\mspace{14mu}\vartheta} \right)^{2}} \right\rbrack}r\mspace{14mu}{dr}\mspace{14mu} d\;\vartheta}}}}} & {{Equation}\mspace{14mu}(1)} \\{\mspace{76mu}{{I_{v} = {{\pi\mspace{14mu} n_{D}i_{s}R^{2}v^{2}} = {N_{E}\mspace{14mu} v^{2}}}},}} & {{Equation}\mspace{14mu}(2)} \\{\mspace{76mu}{I_{R} = {{2\pi\mspace{14mu} n_{D}i_{s}\frac{R^{4}}{4}} = {\frac{1}{2}N_{E}\mspace{14mu}{R^{2}.}}}}} & {{Equation}\mspace{14mu}(3)}\end{matrix}$

The energy law is the sum of two independent components:

-   -   a. I_(v) the position dependence, independent of the radius,        which minimum is positioned at origin, as will have been the        minimum of an infinitesimal point, with N_(E) fluorophores        positioned at the nominal position, and    -   b. I_(R), the radius dependence, which is independent of the        position.

At nominal position, at v=0, the energy is given by equation (3). Theinverse energy law is quadratic with the energy. The same 5%characteristics obtained by an optimal imaging experiment, can be reach,for a shot noise limited system, with only 100 photons, due to thequadratic dependence of the energy as function of the radius.

Measurement of the Size, Position and Ellipticity of a Point-Object

In a second embodiment of the invention, shape parameters may bemeasured: the position, size and ellipticity of a point-object may bemeasured in two dimensions. It is a least common case in measurement ofbiological objects, possibly because it is not possible to perform usingany existing paradigm.

Assuming

-   -   a. a luminous object of elliptical shape, FIG. 2D, with minor-        and major-semi axes a and b respectively,    -   b. the distribution position being given by (0, y_(o)) in        cartesian coordinates, and not in polar coordinates as in the        previous paragraph,    -   c. a uniform density of fluorophores represented as n_(D),    -   d. projecting the half-moons distribution of FIG. 3 ,    -   e. that the half-moon” distributions, case 24 of FIG. 3 , have        been aligned with the direction of the major axis (x-axis in        FIG. 2D);        in first approximation, a quadratic dependence of the intensity        distribution in the direction y, perpendicular to the direction        of the major axis, results. It can be shown that N_(E)=n_(D)        crab, is the total number of emitting fluorophores, where nab is        the area of the ellipse.

An optical system is provided, such as that described in detail in Sirat'185 patent, that is adapted to move the position of a vortex, given as(v, 0), with high accuracy, in the plane. the polar coordinatesnotation, a refers to a radius vector in a specified transverse planewith respect to the z-axis, while b refers to the angle between apredetermined axis (x in FIG. 2C) and the radius. Without loss ofgenerality, due to the symmetry of the problem, we assume that thevortex is positioned on the x-axis, i.e. the angle is zero.

Under the assumption that the distribution is angularly aligned with themajor semi-axis b, the x-axis of FIG. 2D, and displaced by y₀, theintensity of the distribution as a function of position y isquadratically dependent, through a coefficient i_(s), on the distance onaxis y., and independent of x.

The equation of an ellipse is given byx=a cos(t) and y=b sin(t)  Equation (4)at a given (x, y) cartesian position, the intensity is i_(s) (y−y₀)²,i_(s) being the quadratic parameter. The integrated energy along the yaxis, at a given x=A position is given by:I(x)=i _(s) n _(D)∫_(−A) ^(A)(y−y ₀)² dy=i _(s) n _(D)(2A y ₀ ²+3A³),  Equation (5)where A is positioned on an ellipse, A=a sin(t), as stated above.Integration on x yields two terms, the first one, I_(y0), dependingquadratically on y₀:I _(y0)=2i _(s) n _(D) y ₀ ²∫₀ ^(π) a sin(t)dx=2i _(s) n _(D) ab y ₀ ²∫₀^(π) sin² t dt,using Equation (4) again, and its derivative dx=b sin(t)dt. The firstterm, I_(y0), depends on y₀:

$\begin{matrix}{{I_{y\; 0} = {{2\mspace{14mu} i_{s}\mspace{14mu} n_{D}\mspace{14mu}{ab}\mspace{14mu} y_{0}^{2}\frac{\pi}{2}} = {N_{E}\mspace{14mu} i_{s}\mspace{14mu} y_{0}^{2}}}},} & {{Equation}\mspace{14mu}(6)}\end{matrix}$which is the energy which will have been obtained for an infinitesimalpoint, with total number of emitting fluorophores, N_(E), positioned atthe origin. The second term, I_(a), depends on the semi axis valuethrough the equation:I _(a)=3i _(s) n _(D)∫₀ ^(π)(a sin(t))³ dx=3i _(s) n _(D) a ³ b∫ ₀ ^(π)sin⁴ t dt;again using equation (4), and its derivative dx=b sin(t) dt. The secondterm, I_(y0), depends on the semi-axis a through:

$\begin{matrix}{I_{a} = {{3\mspace{14mu} i_{s}\mspace{14mu} n_{D}\mspace{14mu} a^{3}b\frac{3\pi}{8}} = {\frac{9}{8}N_{E}\mspace{14mu} i_{s}\mspace{14mu}{a^{2}.}}}} & {{Equation}\mspace{14mu}(7)}\end{matrix}$The foregoing is an energy depending on the semi-axis of the ellipse, a.A similar measure, using the half-moon” distributions, case 24 of FIG. 3, aligned with the direction of the minor axis (y-axis in FIG. 2D), willyield a term depending on the major axis, and a comparison of these twoterms will allows the measurement of ellipticity.Assessment of the Position of Two Points Positioned at Two DifferentPlanes

In accordance with further embodiments of the present invention, therelative position of two points in space may be measured using thetechniques described herein, even when the two points are positioned attwo different planes.

Referring now to FIG. 4 , the relative position of two points,positioned at two different planes, is a difficult problem, from apractical point of view, and is the heart of one of the most importantmetrology steps in the processing of semiconductors: overlay metrology.Assuming a crescent moon, as in case 02 of FIG. 3 , the locus of thissingular distribution is a spiral 401. The geometric shapes embedded inthis distribution includes spiral shapes but also plurality of pointspositioned along the spiral. If two points are in two different planes403 and 405, but are on the same spiral, the relative position of onepoint relative to the other may be monitored along the spiral and, theposition of two-points at two different planes may be assessed withtheoretically infinite resolution.

Implementation of Methods Using Conical Diffraction

Methods in accordance with embodiments of the present invention may alsoadvantageously be implemented using conical diffraction, employing theteachings of Caron, J., et al, “Conical diffraction illumination opensthe way for low phototoxicity super-resolution imaging,” Cell Adhesion &Migration 8(5): 430-439 (2014), which is incorporated herein byreference.

One embodiment of the invention is now described with reference to FIG.5 . A method, designated generally by numeral 500, is provided fordetermining a value of an intrinsic geometrical parameter of ageometrical feature of a specified dimensionality characterizing aphysical object. To perform the claimed method, CODIM or MetrologyApplication hardware, as described in detail above, is provided (step501) and a singular light distribution is projected (503) onto aphysical object. Light excited by the singular light distribution thathas interacted with the geometrical feature is detected (505) as itimpinges upon a detector. A return energy distribution is identified andquantified (507) at one or more positions of the singular lightdistribution as a quantification of the intrinsic geometrical parameter.Finally, the value of the intrinsic geometrical parameter is determined(509) based on the value of the intrinsic geometrical parameter.

The use of standard deep learning concept on CODIM imaging is nowdescribed with reference to FIG. 7 . A biological object, 20,represented graphically as a bacterium, is illuminated by a series oflocalized singular distributions, 21, detected by a suitable, low noise,imaging detector, 22. A series of microimages 23 (also referred toherein as “μimages,” are recorded and processed by a suitable algorithm,25, to yield a reconstructed, super-resolved image, 10. A deep learningnetwork, including several layers, 11, 12 and 13, is applied on thereconstructed image in a process similar to the one described in FIG. 6. A control system, 24, is used to control and synchronize theacquisition and processing.

The use of a direct deep learning in CODIM imaging is now described withreference to FIG. 8 . The biological object, 20, represented graphicallyas a bacterium, is illuminated by a series of localized singulardistributions, 21, detected by a suitable, low noise, imaging detector,22 also referred to herein as a “detector.” The series of μimages 23 isapplied directly, without reconstruction, to a deep learning layer, 26,which may differ from the layer used in the standard process. Additionallayers, 12 and 13, which may differ from the layer used in the standardprocess are applied on the result of the first layer. A control system,24, is used to control and synchronize the acquisition and processing.

The use of standard deep learning concept on a compound imaging systemis now described with reference to FIG. 9 . A general object 120,represented graphically for a heuristic convenience as a cat, isilluminated by a series of localized singular or regular distributions21, detected by detector, 22. The series of microimages 23, alsoreferred to herein as “intermediate images,” are represented assub-images of a part of the cat picture, but it is only a graphicalartifice. The series of μimages, 23, are recorded and processed by asuitable algorithm 25 to yield a reconstructed, super-resolved image 10.A deep learning network, including several layers, 11, 12 and 13, isapplied on the reconstructed image in a process similar to the onedescribed in FIG. 6 . A control system 24, is used to control andsynchronize the acquisition and processing.

The use of a direct deep learning in a general compound imaging case isnow described with reference to FIG. 10 . The general object 120,represented as a cat, is illuminated by the series of localized singularor regular distributions 21, detected by detector 22. The series ofμimages, 23, or intermediate images, are represented as sub-images of apart of the cat picture, but it is only a graphical artifice. The seriesof μimages is applied directly, without reconstruction, to a deeplearning layer 26, which may differ from the layer used in the standardprocess. Additional layers, 12 and 13, which may differ from the layerused in the standard process, are applied on the result of the firstlayer. Control system, 24 is used to control and synchronize theacquisition and processing.

The use of a controlled direct deep learning in CODIM imaging is nowdescribed with reference to FIG. 11 . The biological object 20,represented graphically as a bacterium, is illuminated by series oflocalized singular distributions 21; the sequence of distributions,including their position, intensity, type, duration and timing isdetermined by the control system, based on external information 30 ofany type, and/or partial or final information from the processing. Thelight distributions, emitted by the object illuminated by the differentlight distributions are detected by a suitable, low noise, imagingdetector, 22. The series of μimages, 23, is applied directly, withoutreconstruction, to a deep learning layer, 26, which may differ from thelayer used in the standard process. Additional layers, 12 and 13, whichmay differ from the layer used in the standard process are applied onthe result of the first layer. The control system, 24, is used tocontrol and synchronize the acquisition and processing and determine thesequence of distributions.

The use of a controlled direct deep learning in a general compoundimaging case is now described with reference to FIG. 12 . A generalobject, 120, represented graphically as a cat, is illuminated by seriesof localized singular or regular distributions, 21. The series ofμimages 23, or intermediate images, are represented as sub-images of apart of the cat picture, but it is only a graphical artifice. Thesequence of distributions, including their position, intensity, type,duration and timing, is determined by the control system, based onexternal information 30 of any type, and/or partial or final informationfrom the processing. The light distributions emitted by the objectilluminated by the different light distributions are detected bydetector 22. The series of μimages is applied directly, withoutreconstruction, to a deep learning layer, 26, which may differ from thelayer used in the standard process. Additional layers, 12 and 13, whichmay differ from the layer used in the standard process are applied onthe result of the first layer. A control system, 24 is used to controland synchronize the acquisition and processing and determine thesequence of distributions.

Classifying a Scene Using Deep Learning

Features generally associated with methods in accordance withembodiments of the present invention include:

-   -   a physical or biological scene, consisting of a plurality of        luminous biological objects, each object being modeled as a        luminous geometrical shape, the geometrical shape being        quantified by an adequate set of parameters,    -   a hardware set-up such as the Imaging Case described in Caron        2014 or the Metrology Case described above;    -   a set of measurements, each measurement consisting of the        projection of a singular distribution at nominal position,        either on a regular or pseudo-regular grid, referred to as “the        imaging case”, or on a custom grid, referred to as “metrological        case”; in the metrological case, the singular distributions are        chosen such that they embed the geometrical shapes, and so, with        nominal parameters, the energy measured will be zero.    -   detecting light, from the singular light distribution that has        interacted with the geometrical feature and that impinges upon a        detector, pixelated or not, the light constituting detected        light;    -   A deep learning or neural network layer, using directly the        detected light as direct input of the neural network layer, able        to classify the scene, as a plurality of shapes, static or        dynamic, the shapes being part of a family of elementary shapes        predetermined or acquired by learning.

In accordance with other embodiments of the invention, direct anddynamic methods may also provide control hardware and a controlprocedure, controlling, dynamically or not, the sequence of measurementsand detected light in order to gather an improved set of measurementsrequired to retrieve the parameters of the shapes, by analysis of theelemental data by standard or deep learning procedures.

Details of Deep Learning Embodiments

In accordance with embodiments of the present invention, methods areprovided that use the specificities and advantages of deep learning foranalyzing imaging and metrology data acquired either by CODIM—ConicalDiffraction Microscopy, as described in Caron 2014, or hardwareimplementing the Metrology Method, described above.

On basis of these methods, new systems for optical measurement ofgeometrical features are described herein, and, more particularly, formethods are described for measurement entailing resolutions greater thanthe diffraction limit imposed by the optics and minimal photon fluxes.

Example

One embodiment of the present invention may be illustrated by thefollowing example; this example is used just as an illustration and doesnot purport to represent a tested experimental case. The luminous objectis assumed to be a biological object and this biological object, forillustration, to be the same Influenza Type A virus (IAV), representedschematically in FIG. 2A; this virus is known to have a typical sizebetween 80 to 120 nm, and it is also known to be in most casesspherical, but it may also be in some case filamentous. Badham et al.,“Filamentous Influenza Viruses,” Current clinical microbiology reports,vol. 3, pp. 155-61 (2016), incorporated herein by reference, state that“The biological significance of IAV morphology in human clinicalinfections is a subject of great interest.” The IAV virus is assumed tobe marked uniformly in all its volume with some adequate fluorescentproteins. Several models may be used to describe the virus; the virus isdescribed as a point-object, modeled as an ellipse (FIG. 2D), withparameters being the half-minor axis, a, the half-major axis, b, and theangle of the major axis, θ, relative to a Cartesian referential.

In accordance with embodiments of the present invention, dynamic changesin the position and morphology of a virus are assessed, in order,potentially, to assess the virulence of the virus.

CODIM Embodiments

For imaging using deep learning, the CODIM system described in Caron2014 may be applied, while, for metrology, the set-up described in theMetrology Application is preferred.

In accordance with embodiments of the present invention, methods areprovided that use the specificities and advantages of deep learning foranalyzing imaging and metrology data acquired either by CODIM—ConicalDiffraction Microscopy, as described in Caron 2014.

Neural Network Embodiments

Spiliotis et al., “Priming for destruction: septins at the crossroads ofmitochondrial fission and bacterial autophagy,” EMBO Reports, vol. 17,pp. 935-37 (2016) (“Spiliotis 2016”), incorporated herein by reference,taught:

-   -   Mitochondria are essential organelles for cell survival,        programmed cell death, and autophagy. They undergo cycles of        fission and fusion, which are subverted by infectious pathogens        and altered in many human diseases.        Spiliotis 2016, at 935.

Pagliuso et al., “A role for septin 2 in Drp1-mediated mitochondrialfission”, EMBO Reports, vol. 17, pp. 857-73 (2016) (“Pagliuso 2016”),incorporated herein by reference, showed that,

-   -   among the multiple mechanisms that appear to regulate        mitochondrial fission . . . the Septin 2 has an important role        by mediating mitochondrial constriction.        Pagliuso 2016, Abstract, emphasis added.        This research used a BioAxial CODIM system to acquire images of        the mitochondria, by acquiring the data and reconstruction the        images.

The reconstructed images of mitochondria can be used as the input of adeep learning network, to gather information on biological status, asshown in FIG. 10 ; alternatively, the gathered data can be fed directlyto the deep learning network as shown in FIG. 11 .

The embodiments of the invention described herein are intended to bemerely exemplary; variations and modifications will be apparent to thoseskilled in the art. All such variations and modifications are intendedto be within the scope of the present invention as defined in theappended claims.

I claim:
 1. A method for determining a value of a non-positional intrinsic geometrical parameter of a geometrical feature of a specified dimensionality characterizing a physical object, the method comprising: a. projecting a singular light distribution characterized by a first wavelength and a position of singularity onto the physical object; b. detecting light excited by the singular light distribution that has interacted with the geometrical feature and that impinges upon a detector, the light constituting detected light; c. identifying and quantifying a return energy distribution at one or more positions of the singular light distribution as a quantification of the intrinsic geometrical parameter; and d. determining the value of the intrinsic geometrical parameter based on the value of the intrinsic geometrical parameter.
 2. A method, in which successive applications of the method of claim 1 are performed, in such a manner as to retrieve a single intrinsic geometrical parameter or a set of intrinsic geometrical parameters.
 3. A method for determining a value of a non-positional intrinsic geometrical parameter of a geometrical feature of a specified dimensionality characterizing a physical object, the intrinsic geometrical parameter being size of a point-object, the method comprising: a. projecting a singular light distribution characterized by a first wavelength and a position of singularity onto the physical object b. detecting light excited by the singular light distribution that has interacted with the geometrical feature and that impinges upon a detector, the light constituting detected light; c. identifying and quantifying a return energy distribution at one or more positions of the singular light distribution as a quantification of the intrinsic geometrical parameter; and d. determining the value of the intrinsic geometrical parameter based on the value of the return energy distribution.
 4. A method in accordance with claim 3, wherein detecting light includes employing a pixelated detector.
 5. A method for determining a value of an intrinsic geometrical parameter of a geometrical feature of a specified dimensionality characterizing a physical object, the intrinsic geometrical parameter being ellipticity of a point-object, the method comprising: a. projecting a singular light distribution characterized by a first wavelength and a position of singularity onto the physical object b. detecting light excited by the singular light distribution that has interacted with the geometrical feature and that impinges upon a detector, the light constituting detected light; c. identifying and quantifying a return energy distribution at one or more positions of the singular light distribution as a quantification of the intrinsic geometrical parameter; and d. determining the value of the intrinsic geometrical parameter based on the value of the return energy distribution.
 6. A method in accordance with claim 5, wherein detecting light includes employing a pixelated detector.
 7. A method for determining a value of a non-positional intrinsic geometrical parameter of a geometrical feature of a specified dimensionality characterizing a physical object, the intrinsic geometrical parameter characterizing a model shape of the geometrical feature, the method comprising: a. projecting a singular light distribution characterized by a first wavelength and a position of singularity onto the physical object b. detecting light excited by the singular light distribution that has interacted with the geometrical feature and that impinges upon a detector, the light constituting detected light; c. identifying and quantifying a return energy distribution at one or more positions of the singular light distribution as a quantification of the intrinsic geometrical parameter; and d. determining the value of the intrinsic geometrical parameter based on the value of the return energy distribution.
 8. A method in accordance with claim 7, wherein detecting light includes employing a pixelated detector. 